MK8S-184 - Add restart script to react when control plane ingress change by ChengYanJin · Pull Request #4822 · scality/metalk8s

ChengYanJin · 2026-03-17T17:35:40Z

Component: Salt

Context:
This PR adds an automatic restart mechanism for the oauth2-proxy deployments when the CA certificate secret changes.

When OIDC authentication is enabled, the oauth2-proxy pods use a k8s-sidecar to sync the CA certificate from a Kubernetes secret. However, when the CA secret is updated (e.g., certificate rotation), the oauth2-proxy process keeps using the old CA loaded in memory, causing authentication failures.

Summary:
Leverage the k8s-sidecar's SCRIPT feature to run a Python script whenever the CA secret changes. The script:

Computes a SHA256 hash of the CA files on disk
Compares it with the previously stored hash
If changed, triggers a rolling restart of the deployment via the Kubernetes API (same as kubectl rollout restart)
On initial startup, saves the hash and skips restart to avoid a restart loop

Acceptance criteria:

When the CA secret data changes, the oauth2-proxy pods (prometheus and alertmanager) are automatically restarted via a rolling restart.

[Closes: #MK8S-184]

bert-e · 2026-03-17T17:35:44Z

Hello chengyanjin,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-17T17:35:51Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-03-18T09:21:59Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

salt/metalk8s/addons/nginx-ingress-control-plane/deployed/tls-secret.sls

salt/tests/unit/formulas/fixtures/salt.py

TeddyAndrieux

I do not really like using script in configmap like that 😕 but I don't have any quick easy way to handle it so ... let's go like this for now

salt/metalk8s/addons/prometheus-operator/deployed/oidc-proxy-restart-script.sls

salt/tests/unit/formulas/fixtures/salt.py

salt/metalk8s/addons/prometheus-operator/deployed/files/restart-on-ca-change.py

salt/metalk8s/addons/prometheus-operator/deployed/oidc-proxy-alertmanager.sls

salt/tests/unit/scripts/test_restart_on_ca_change.py

bert-e · 2026-03-19T15:40:18Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

bert-e · 2026-03-23T11:33:09Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

…hebang and defaultMode

claude · 2026-03-24T14:54:36Z

salt/tests/unit/scripts/test_restart_on_ca_change.py

@@ -0,0 +1,208 @@
+"""Tests for the restart-on-ca-change.py script."""


Missing __init__.py in salt/tests/unit/scripts/. Every other test subdirectory under salt/tests/unit/ has one (e.g. formulas/, mocks/). Without it, pytest with the default import mode may not discover these tests.

— Claude Code

claude · 2026-03-24T14:54:37Z

salt/metalk8s/addons/prometheus-operator/deployed/oidc-proxy-prometheus.sls

+              - name: restart-script
+                configMap:
+                  name: oidc-proxy-restart-script
+                  defaultMode: "0555"


defaultMode should be an integer (0555), not a string ("0555"). Kubernetes accepts strings here but interprets them as decimal — "0555" becomes octal 01053 (decimal 555), which is not what you want. Use an unquoted integer instead.

— Claude Code

claude · 2026-03-24T14:54:38Z

salt/metalk8s/addons/prometheus-operator/deployed/oidc-proxy-alertmanager.sls

+              - name: restart-script
+                configMap:
+                  name: oidc-proxy-restart-script
+                  defaultMode: "0555"


Same issue: defaultMode should be an integer, not a string. "0555" is interpreted as decimal 555 (octal 01053), not as octal 0555.

— Claude Code

claude · 2026-03-24T14:55:03Z

defaultMode: "0555" in both deployment templates (oidc-proxy-prometheus.sls:111, oidc-proxy-alertmanager.sls:112) uses a quoted string instead of an integer. In YAML 1.1 (PyYAML/Salt), unquoted 0555 is parsed as octal (365 decimal = rwxr-xr-x), but "0555" is a string that may be coerced to decimal 555 (octal 01053) or rejected by the K8s API. Remove the quotes to get the intended permissions.
- Change defaultMode: "0555" to defaultMode: 0555 in both files
Missing __init__.py in salt/tests/unit/scripts/ — all other test subdirectories under salt/tests/unit/ have one, and without it pytest may not discover these tests.
- Add an empty __init__.py to salt/tests/unit/scripts/
  
  Review by Claude Code

claude · 2026-03-24T15:09:42Z

salt/metalk8s/addons/prometheus-operator/deployed/oidc-proxy-prometheus.sls

+              - name: restart-script
+                configMap:
+                  name: oidc-proxy-restart-script
+                  defaultMode: "0555"


defaultMode in Kubernetes expects an integer, not a string. 0555 quoted is a YAML string and may be passed as-is to the API, where it could be rejected or misinterpreted. Use the unquoted octal literal 0555 (YAML interprets this as decimal 365) or just 365. The existing thanos-chart.sls uses defaultMode: 420 (integer) as a reference.

— Claude Code

claude · 2026-03-24T15:09:47Z

salt/metalk8s/addons/prometheus-operator/deployed/oidc-proxy-alertmanager.sls

+              - name: restart-script
+                configMap:
+                  name: oidc-proxy-restart-script
+                  defaultMode: "0555"


Same issue: defaultMode should be an integer, not a string.

— Claude Code

claude · 2026-03-24T15:09:56Z

defaultMode: "0555" is a YAML string but the Kubernetes API expects an integer for this field. This could cause the manifest to be rejected or the mode to be misinterpreted. Use defaultMode: 0555 (unquoted octal, = 365 decimal) or defaultMode: 365. Affects both oidc-proxy-prometheus.sls and oidc-proxy-alertmanager.sls.
- Remove the quotes around 0555 in both deployment specs

Review by Claude Code

claude · 2026-03-24T15:22:35Z

salt/metalk8s/addons/prometheus-operator/deployed/oidc-proxy-prometheus.sls

+              - name: restart-script
+                configMap:
+                  name: oidc-proxy-restart-script
+                  defaultMode: "0555"


defaultMode expects an integer, not a string. "0555" will be passed as the string "0555" to the Kubernetes API, which expects a decimal integer. Octal 0555 = decimal 365. The rest of the codebase uses unquoted decimal integers (e.g., defaultMode: 420 for 0644).

— Claude Code

claude · 2026-03-24T15:22:37Z

salt/metalk8s/addons/prometheus-operator/deployed/oidc-proxy-alertmanager.sls

+              - name: restart-script
+                configMap:
+                  name: oidc-proxy-restart-script
+                  defaultMode: "0555"


Same defaultMode issue — should be an unquoted decimal integer 365 instead of the string "0555".

— Claude Code

claude · 2026-03-24T15:22:45Z

defaultMode: "0555" in both oidc-proxy-prometheus.sls and oidc-proxy-alertmanager.sls passes a string instead of an integer to the Kubernetes API. The rest of the codebase uses unquoted decimal integers (e.g., 420 for octal 0644). Use defaultMode: 365 (decimal for octal 0555).
- oidc-proxy-prometheus.sls:111
- oidc-proxy-alertmanager.sls:112

Review by Claude Code

TeddyAndrieux

LGTM

bert-e · 2026-03-25T09:29:55Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

ChengYanJin · 2026-03-25T09:30:47Z

/approve

bert-e · 2026-03-25T09:30:56Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

MonPote

LGTM

bert-e · 2026-03-25T13:46:37Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/133.0

The following branches will NOT be impacted:

development/123.0
development/124.0
development/124.1
development/125.0
development/126.0
development/127.0
development/128.0
development/129.0
development/130.0
development/131.0
development/132.0
development/2.0
development/2.1
development/2.10
development/2.11
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2026-03-25T16:02:27Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/133.0

The following branches have NOT changed:

development/123.0
development/124.0
development/124.1
development/125.0
development/126.0
development/127.0
development/128.0
development/129.0
development/130.0
development/131.0
development/132.0
development/2.0
development/2.1
development/2.10
development/2.11
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

Please check the status of the associated issue MK8S-184.

Goodbye chengyanjin.

ChengYanJin force-pushed the feature/MK8S-184-restart-script branch from 1a109d4 to 103f7c0 Compare March 18, 2026 09:21

ChengYanJin added 2 commits March 18, 2026 10:33

MK8S-184 - Add restart script to react when control plane ingress change

6821603

MK8S-184 - Add label to control plane ingress secret

52eb865

ChengYanJin force-pushed the feature/MK8S-184-restart-script branch from 103f7c0 to 8498223 Compare March 18, 2026 09:33

ChengYanJin added 2 commits March 18, 2026 10:45

MK8S-184 - Add a mock for Salt cp.get_file_str

2a4c95e

MK8S-184 - Add the new sls file in the salt tree

092df26

ChengYanJin force-pushed the feature/MK8S-184-restart-script branch from 8498223 to 092df26 Compare March 18, 2026 09:45

ChengYanJin marked this pull request as ready for review March 18, 2026 10:19

ChengYanJin requested a review from a team as a code owner March 18, 2026 10:19

MK8S-184 - Add test for restart script

42b899c

ChengYanJin force-pushed the feature/MK8S-184-restart-script branch from a6bb1c3 to 42b899c Compare March 18, 2026 10:59

MonPote reviewed Mar 18, 2026

View reviewed changes

salt/metalk8s/addons/nginx-ingress-control-plane/deployed/tls-secret.sls Show resolved Hide resolved

salt/tests/unit/formulas/fixtures/salt.py Outdated Show resolved Hide resolved

MK8S-197 - improve the mock

bcc8970

TeddyAndrieux reviewed Mar 19, 2026

View reviewed changes

ChengYanJin force-pushed the feature/MK8S-184-restart-script branch 3 times, most recently from 1db7250 to a7ff0d9 Compare March 23, 2026 13:48

ChengYanJin added 4 commits March 23, 2026 14:52

MK8S-184 - Improve the Python script

09d35e8

MK8S-184 - Rename POD_NAMESPACE to DEPLOYMENT_NAMESPACE

edbd810

MK8S-184 - Remove shell wrapper, execute Python script directly via s…

3b1e212

…hebang and defaultMode

MK8S-184 - Move the test to a scripts folder

ab2f406

ChengYanJin force-pushed the feature/MK8S-184-restart-script branch from a7ff0d9 to ab2f406 Compare March 23, 2026 13:52

claude bot reviewed Mar 24, 2026

View reviewed changes

MK8S-184 - Use modern Python library pathlib.Path instead of os.path

60f0f10

ChengYanJin force-pushed the feature/MK8S-184-restart-script branch from 17e78ea to 60f0f10 Compare March 24, 2026 15:06

claude bot reviewed Mar 24, 2026

View reviewed changes

MK8S-184 - Add __init__ in the unit/script folder

377a07a

claude bot reviewed Mar 24, 2026

View reviewed changes

TeddyAndrieux approved these changes Mar 25, 2026

View reviewed changes

MonPote approved these changes Mar 25, 2026

View reviewed changes

bert-e merged commit 2ecf9ec into development/133.0 Mar 25, 2026
31 checks passed

bert-e deleted the feature/MK8S-184-restart-script branch March 25, 2026 16:02

		@@ -0,0 +1,208 @@
		"""Tests for the restart-on-ca-change.py script."""

Conversation

ChengYanJin commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Mar 17, 2026

Hello chengyanjin,

Uh oh!

bert-e commented Mar 17, 2026

Waiting for approval

Uh oh!

bert-e commented Mar 18, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

TeddyAndrieux left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bert-e commented Mar 19, 2026

Waiting for approval

Uh oh!

bert-e commented Mar 23, 2026

Waiting for approval

Uh oh!

claude bot Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot commented Mar 24, 2026

Uh oh!

claude bot Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot commented Mar 24, 2026

Uh oh!

claude bot Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot commented Mar 24, 2026

Uh oh!

TeddyAndrieux left a comment

Choose a reason for hiding this comment

Uh oh!

bert-e commented Mar 25, 2026

Waiting for approval

Uh oh!

ChengYanJin commented Mar 25, 2026

Uh oh!

bert-e commented Mar 25, 2026

Waiting for approval

Uh oh!

MonPote left a comment

Choose a reason for hiding this comment

Uh oh!

bert-e commented Mar 25, 2026

In the queue

Uh oh!

bert-e commented Mar 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

ChengYanJin commented Mar 17, 2026 •

edited

Loading